Multimodal Temporal Patterns for the Analysis of User’s Involvement in Affective Interaction with Virtual Agents

Given the growing interest in developing embodied virtual agents with multimodal communication and emotional expression abilities, the issue of user’s involvement is a relevant topic to take into account in determining how to assess and interpret the quality of user-agent affective interaction. Main goal of this paper is the definition of a methodology for the analysis of user-agent interaction synchrony considered as an index of user’s involvement. The proposed approach is based on recent advances in communication psychology, which on the one hand show the importance of considering the hidden temporal organization underlying communicative interaction and on the other hand provide a specific methodology for the structural analysis of the interactive flow (analysis of intraand interindividual multimodal behavioral patterns through Theme software). From a theoretical point of view, the crucial assumption is that the more synchronic interactions are, the more pleasant and fulfilling they are experienced, and consequently more related to positively valenced emotional states. Our main objective is to tune and to test this methodology (typically used in analyzing human-human communication exchanges) within user-agent interaction, in order to detect interactive temporal patterns of actions in affective interactions with virtual agents. This approach has been developed within the European project MYSELF, where we are a preliminary evaluation study of an interactive pedagogical agent by combining self-report measures, physiological measures and multimodal behavioral patterns approach is being carried out.